Artificial intelligence in health care: accountability and safety

The prospect of patient harm caused by the decisions made by an artificial intelligence-based clinical tool is something to which current practices of accountability and safety worldwide have not yet adjusted. We focus on two aspects of clinical artificial intelligence used for decision-making: moral accountability for harm to patients; and safety assurance to protect patients against such harm. Artificial intelligence-based tools are challenging the standard clinical practices of assigning blame and assuring safety. Human clinicians and safety engineers have weaker control over the decisions reached by artificial intelligence systems and less knowledge and understanding of precisely how the artificial intelligence systems reach their decisions. We illustrate this analysis by applying it to an example of an artificial intelligence-based system developed for use in the treatment of sepsis. The paper ends with practical suggestions for ways forward to mitigate these concerns. We argue for a need to include artificial intelligence developers and systems safety engineers in our assessments of moral accountability for patient harm. Meanwhile, none of the actors in the model robustly fulfil the traditional conditions of moral accountability for the decisions of an artificial intelligence system. We should therefore update our conceptions of moral accountability in this context. We also need to move from a static to a dynamic model of assurance, accepting that considerations of safety are not fully resolvable during the design of the artificial intelligence system before the system has been deployed

Enhancing Covid-19 Decision-Making by Creating an Assurance Case for Simulation Models

Simulation models have been informing the COVID-19 policy-making process. These models, therefore, have significant influence on risk of societal harms. But how clearly are the underlying modelling assumptions and limitations communicated so that decision-makers can readily understand them? When making claims about risk in safety-critical systems, it is common practice to produce an assurance case, which is a structured argument supported by evidence with the aim to assess how confident we should be in our risk-based decisions. We argue that any COVID-19 simulation model that is used to guide critical policy decisions would benefit from being supported with such a case to explain how, and to what extent, the evidence from the simulation can be relied on to substantiate policy conclusions. This would enable a critical review of the implicit assumptions and inherent uncertainty in modelling, and would give the overall decision-making process greater transparency and accountability.Comment: 6 pages and 2 figure

Safe Reinforcement Learning for Sepsis Treatment

Sepsis, a life-threatening illness, is estimated to be the primary cause of death for 50,000 people a year in the UK and many more worldwide. Managing the treatment of sepsis is very challenging as it is frequently missed, at an early stage, and the optimal treatment is not yet clear. There are promising attempts to use Reinforcement Learning (RL) to learn the optimal strategy to treat sepsis patients, especially for the administration of intravenous fluids and vasopressors. However, RL agents only take the current state of patients into account when recommending the dosage of vasopressors. This is inconsistent with current clinical safety practice in which the dosage of vasopressors is increased or decreased gradually. A sudden major change of the dosage might cause significant harm to patients and as such is considered unsafe in sepsis treatment. In this paper, we have adapted one of the deep RL methods published previously and evaluated it to assess whether it has this kind of sudden major change when recommending the vasopressor dosage. Then, we have modified this method to address the above safety constraint and learnt a safer policy by incorporating current clinical knowledge and practice

Enhancing COVID-19 decision making by creating an assurance case for epidemiological models

When the UK government was first confronted with the very real threat of a COVID-19 pandemic, policy-makers turned quickly, and initially almost exclusively, to scientific data provided by epidemiological models. These models have had a direct and significant influence on the policies and decisions, such as social distancing and closure of schools, which aim to reduce the risk posed by COVID-19 to public health.1 The models suggested that depending on the strategies chosen, the number of deaths could vary by hundreds of thousands. From a safety engineering perspective, it is clear that the data generated by epidemiological models are safety critical, and that, therefore, the models themselves should be regarded as safety-critical systems

Tracking proactive interference in visual memory

© 2022 The Authors. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: https://doi.org/10.3389/fpsyg.2022.896866The current contents of visual working memory can be disrupted by previously formed memories. This phenomenon is known as proactive interference, and it can be used to index the availability of old memories. However, there is uncertainty about the robustness and lifetime of proactive interference, which raises important questions about the role of temporal factors in forgetting. The present study assessed different factors that were expected to influence the persistence of proactive interference over an inter-trial interval in the visual recent probes task. In three experiments, participants encoded arrays of targets and then determined whether a single probe matched one of those targets. On some trials, the probe matched an item from the previous trial (a “recent negative”), whereas on other trials the probe matched a more distant item (a “non-recent negative”). Prior studies have found that recent negative probes can increase errors and slow response times in comparison to non-recent negative probes, and this offered a behavioral measure of proactive interference. In Experiment 1, factors of array size (the number of targets to be encoded) and inter-trial interval (300 ms vs. 8 s) were manipulated in the recent probes task. There was a reduction in proactive interference when a longer delay separated trials on one measure, but only when participants encoded two targets. When working memory capacity was strained by increasing the array size to four targets, proactive interference became stronger after the long delay. In Experiment 2, the inter-trial interval length was again manipulated, along with stimulus novelty (the number of stimuli used in the experiment). Proactive interference was modestly stronger when a smaller number of stimuli were used throughout the experiment, but proactive interference was minimally affected by the inter-trial interval. These findings are problematic for temporal models of forgetting, but Experiment 3 showed that proactive interference also resisted disruption produced by a secondary task presented within the inter-trial interval. Proactive interference was constantly present and generally resilient to the different manipulations. The combined data suggest a relatively durable, passive representation that can disrupt current working memory under a variety of different circumstances.This work was funded by the Experimental Psychology Society’s Small Grant Scheme. Open Access Publication Fees were covered

Predicting Progression of Type 2 Diabetes Using Primary Care Data with the Help of Machine Learning

Type 2 diabetes is a life-long health condition, and as it progresses, A range of comorbidities can develop. The prevalence of diabetes has increased gradually, and it is expected that 642 million adults will be living with diabetes by 2040. Early and proper interventions for managing diabetes-related comorbidities are important. In this study, we propose a Machine Learning (ML) model for predicting the risk of developing hypertension for patients who already have Type 2 diabetes. We used the Connected Bradford dataset, consisting of 1.4 million patients, as our main dataset for data analysis and model building. As a result of data analysis, we found that hypertension is the most frequent observation among patients having Type 2 diabetes. Since hypertension is very important to predict clinically poor outcomes such as risk of heart, brain, kidney, and other diseases, it is crucial to make early and accurate predictions of the risk of having hypertension for Type 2 diabetic patients. We used Naïve Bayes (NB), Neural Network (NN), Random Forest (RF), and Support Vector Machine (SVM) to train our model. Then we ensembled these models to see the potential performance improvement. The ensemble method gave the best classification performance values of accuracy and kappa values of 0.9525 and 0.2183, respectively. We concluded that predicting the risk of developing hypertension for Type 2 diabetic patients using ML provides a promising stepping stone for preventing the Type 2 diabetes progression

Utility of the new Movement Disorder Society clinical diagnostic criteria for Parkinson's disease applied retrospectively in a large cohort study of recent onset cases

Objective: To examine the utility of the new Movement Disorder Society (MDS) diagnostic criteria in a large cohort of Parkinson's disease (PD) patients. Methods: Recently diagnosed (<3.5 years) PD cases fulfilling United Kingdom (UK) brain bank criteria in Tracking Parkinson's, a UK multicenter prospective natural history study were assessed by retrospective application of the MDS criteria. Results: In 2000 cases, 1835 (91.7%) met MDS criteria for PD, either clinically established (n = 1261, 63.1%) or clinically probable (n = 574, 28.7%), leaving 165 (8.3%) not fulfilling criteria. Clinically established cases were significantly more likely to have limb rest tremor (89.3%), a good l-dopa response (79.5%), and olfactory loss (71.1%), than clinically probable cases (60.6%, 44.4%, and 34.5% respectively), but differences between probable PD and ‘not PD’ cases were less evident. In cases not fulfilling criteria, the mean MDS UPDRS3 score (25.1, SD 13.2) was significantly higher than in probable PD (22.3, SD 12.7, p = 0.016) but not established PD (22.9, SD 12.0, p = 0.066). The l-dopa equivalent daily dose of 341 mg (SD 261) in non-PD cases was significantly higher than in probable PD (250 mg, SD 214, p < 0.001) and established PD (308 mg, SD 199, p = 0.025). After 30 months' follow-up, 89.5% of clinically established cases at baseline remained as PD (established/probable), and 86.9% of those categorized as clinically probable at baseline remained as PD (established/probable). Cases not fulfilling PD criteria had more severe parkinsonism, in particular relating to postural instability, gait problems, and cognitive impairment. Conclusion: Over 90% of cases clinically diagnosed as early PD fulfilled the MDS criteria for PD. Those not fulfilling criteria may have an atypical parkinsonian disorder or secondary parkinsonism that is not correctly identified by the UK Brain Bank criteria, but possibly by the new criteria

The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

Established approaches to assuring safety-critical systems and software are difficult to apply to systems employing ML where there is no clear, pre-defined specification against which to assess validity. This problem is exacerbated by the opaque nature of ML where the learnt model is not amenable to human scrutiny. XAI methods have been proposed to tackle this issue by producing human-interpretable representations of ML models which can help users to gain confidence and build trust in the ML system. However, little work explicitly investigates the role of explainability for safety assurance in the context of ML development. This paper identifies ways in which XAI methods can contribute to safety assurance of ML-based systems. It then uses a concrete ML-based clinical decision support system, concerning weaning of patients from mechanical ventilation, to demonstrate how XAI methods can be employed to produce evidence to support safety assurance. The results are also represented in a safety argument to show where, and in what way, XAI methods can contribute to a safety case. Overall, we conclude that XAI methods have a valuable role in safety assurance of ML-based systems in healthcare but that they are not sufficient in themselves to assure safety

Developing a Safety Case for Electronic Prescribing

It is now recognised that Health IT systems can bring benefits to healthcare, but they can also introduce new causes of risks that contribute to patient harm. This paper focuses on approaches to modelling and analysing potential causes of medication errors, particularly those arising from the use of Electronic Prescribing. It sets out a systematic way of analysing hazards, their causes and consequences, drawing on the expertise of a multidisciplinary team. The analysis results are used to support the development of a safety case for a large-scale Health IT system in use in three teaching hospitals. The paper shows how elements of the safety case can be updated dynamically. We show that it is valuable to use the dynamically updated elements to inform clinicians about changes in risk, and thus prompt changes in practice to mitigate the risks